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ABSTRACT 

This report describes how to apply techniques that 
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SUMMARY 



Problem 

This manual presents techniques and considerations for reduc- 
ing the reading level of Air Force training materials so as to improve 
the capability of these materials to transfer information to the student. 

Approach 

A review of available readability/ comprehensibility measure- 
ment techniques and previous experimental studies into methods for 
developing such techniques is presented. As the result of this review, 
techniques for reading level determination and methods for increas- 
ing the readability/ comprehensibility of textual materials are discuss- 
ed. The application of the techniques is demonstrated, and practical 
considerations, including cost/ effectiveness determination, for the Air 
Force are considered. Methods for presenting subject matter mate- 
rial, other than through written prose, are described. In order to 
determine the effectiveness of a modified technique, performance fol- 
lowups are suggested. Various experimental techniques and testing 
procedures for this purpose are discussed. 

Results 

Several techniques for measuring readability are described 
in detail, and methods for their use and application are fully deline- 
ated. The readability techniques included are: cloze, Flesch, FOR- 
CAST, SMOG, and the Automated Readability .Index (ARI). A quantita- 
tive approach for determining the cost/effectiveness of a subject mat- 
ter presentation technique is developed and described. Instructions 
are provided for presenting material through auditory cassettes and 
for preparing illustrations of various types. Techniques for simplify- 
ing material are explained. 

Conclusions 



Application of the various methods and techniques described 
will reduce training time and costs and will serve to increase the lev- 
el of comprehension by students who are exposed to various training 
materials. 



ERIC 



1, 

/ •■ 
/ 



TABLE OF CONTENTS 

Page 

I. INTRODUCTION 7 

Techniques for Measuring Readability 7 

Early (1920s) Studies into Analytic Methods 8 

Studies During the 1930s 10 

Studies of the 1940s and Early 1950s 12 

Recent Studies 14 

The Flesch Method 17 

The Cloze Procedure , . 18 

I'he Dale and Chall Method 20 

Reading Level Determination 20 

n. APPLICATIONS FOR INCREASING RELIABILITY 23 

Gains to be Anticipated , 23 

Time for Comprehension ^ 24 

Logical and Precise Writing , , , , 24 

Word and Sentence Simplification 28 

Repetition-Redundancy 31 

Textual Supplementation 31 

Auditory Supplementation 31 

Illustration Supplementation 33 

P''lowcharts and Tables , 35 

Graphs 39 

Schematic Diagrams, , . , , 1+3 

III. "NONA JALYTIC"METH0DS FOR EVALUATING TEXTUAL 

METHODS 49 

Design of Materials Evaluations 1+9 

'Single Group Methods 50 

r;<....ntrol Group Studies 50 

Factorial Design 51 

Conduct of Nonanaiytic Studies 53 

Characteristics of Useful Moasur^e of Performance . ... , 51+ 



3 



PB,^e 



Examples of Dependent Variables 55 

Written Tests 56 

Criterion and Norm Referenced Testing 56 

Performance Tests *. 57 

The Evaluative Milieu 58 

Additional Reading List 59 

REFERENCES 60 

APPENDIX A How to Perform a Cloze Analysis 63 

APPENDIX B Examples of Before and After Material and 

Automated Reading Index (ARI) and Grade 
Level (GL) Analyses 68 

APPENDIX C How to Perform a Flesch Analysis 71 

APPENDIX D How to Perform the FORCAST Analysis 77 

APPENDIX E How to Perform the SMOG Analysis 78 

APPENDIX F Glossary of Readability Measures 81 



ERIC 



4 



LIST OF ILLUSTRATIONS 

B^igure Page 

1 Sample of instructions in prose 35 

2 Example of flowchart to represent instructions of 

Exhibit I 36 

3 Example of tabular presentation of instructions 33 

4 Weight as a function of height for men and women between 

the ages of 18 and 64 i+l 

5 Annual mean income of males (25 years and older) in 

1956 and 1965 as a function of years of education 1+2 

6 Percentage of United States' population completing 

various years of schooling, 1966 

7 Example of schematic incorporating comprehensibility 

principles U6 



ERIC 



!• INTRODUCTION 



The mismatch between the reading grade level of military per- 
sonnel and the readability level of the material they read has been 
found to be quite vast in some instances (i. e. , four to eight grade 
levels). This manual presents teclxniques which can be used to im- 
prove arid measure the readability/ iatelligibility of Air Force read- 
ing materials and, accordingly, reduce the gap between the reading 
grade levels of the readers and the material they read. 

This manual is intended to serve as a source document for use 
by Air Force training and development personnel, as well as others 
who prepare reading material for the Air Force, The techniques 
preseiited are described historically, evaluated, and demonstrated. 
Methods for evaluating the comprehensibility of reading material are 
discussed. Finally, several practical supplements are presented. 
These contain information regarding auditory, pictorial, and multi- 
sensory presentation of information. Such considerations are impor- 
tant to the preparation of materials used in training some types of per- 
sonnel having limited reading skills. 

Techniques for Measuring Readability 

The readability of written material is linked to the ease of un- 
derstanding the material and to the speed at which the printed matter 
can be read. A large number of methods are available for assessing 
readability. In this manual, we group them under: (1) analytic, and 
(2) nonanalytic methods. Analytic methods include any method which 
involves isolation of the various characteristics of the textual materi- 
al itself. Nonanalytic methods are based on some form of measure- 
ment of what has been learned by the reader from the text. 

Most of the analytic methods are based on come formula which 
considers the structural attributes (e.g., sentence length, word length, 
sentence complexity, use of familiar words) of the written material. 
Such formulas provide a quantitative index of the readability of the ma- 
terial. Under analytic methods, we also include the '^cloze^' technique 
(Appendix A). 
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Although the readability formulas are practical in many situ- 
ations, they also suffer from certain drawbacks that can be avoided 
through the use of the nonanalytic approaches. Specifically, while 
the analytic methods provide an index of various attributes of written 
material (such as difficulty, reading grade level, interest), they tell 
us little about how much is learned from reading the text. The non- 
analytic methods provide such information. 

Early (1920s) Studies into Analytic Methods 

Initial analytic efforts to measure readability objectively were 
sparked by the difficulty experienced by teaches when they attempted 
to select the most appropriate texts for their classes. Science teach- 
ers, in particular, found that the textbooks intended for classroom use 
contained a vocabulary level which was beyond that of the students. Con- 
sequently, objective methods were developed for determininff the vocabu- 
lary difficulty of textbooks, so that those books with exceptionally diffi- 
cult vocabulary could be avoided. 

One early technique was developed in 1923 by Lively and Pressey. 
They developed an index number to reflect vocabulary difficulty. In order 
to obtain the index, a sample of 1, 000 words, evenly distributed through- 
out a text, was taken. T\en reference was made to a list of the 10, 000 
most commonly occurring words in the English language. This list was 
compiled by Thorndike in 1921. Values were assigned to each word in 
the sample, such that if the word appeared in the most common thousand 
it received an index value of 10; those words in the second thousand re- • 
ceived a value of 9, and so on down through the last thousand -word block 
in the list. Words not appearing on the list were assigned an index value 
of zero. The difficulty of the vocabulary of the text under consideration 
was computed by obtaining the median of the index values assigned to 
each word. The resulting vocabulary difficulty indices agreed substanti- 
ally with the judged difficulty of a wide variety of reading selections (from 
second grade reading to college level). However, such a readability de- 
termination technique possesses a number of drawbacks. These include 
considerations of: (1) the appropriateness of the 10, 000 word reference 
list, (2) the small size of the sample (samples of more than 1, 000 words 
from an entire book may be more appropriate), (3) prior experience with 
the subject matter area (readability may be easy, despite difficult vocabu- 
lary, because of familiarity with the subject matter, or readability may be 
difficult, despite easy vocabulary, because of a lack of familiarity with the 
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subject mattei:), (4) difficulty of the concepts, (5) the reader's gen- 
eral intejligence, and (6) his interest in the subject matter, 

Washburne and Vogel, working in the lUinois school system, 
generated norms for use with the Lively and Pressey method. These 
norms were based on calculated Lively and Pressey index values for 
700 books that were reported as having been read and liked ouring the 
preceding year by at least 25 students. They also determined the dif- 
ficulty level for 150 of the 700 books. To do this, books read by lower 
grade level students and by liighor grade level students were separated. 
Several factors were found that related to the criterion of grade level 
(these factors could distinguish those books read by. lower grade pupils 
from those read by higher ^rade pupils). These factors correlated sig- 
nificantly with the difficulty of the textual material. Four of these fac- 
tors were set into a formuia for predicting readability (Y). The four 
factors were: number of di<:ferent wor-.ls appearing (x^), number of oc- 
currences of prepositions (x2), nvmb'ir of words not on Thorndike's 
list of 10, 000 (xg), and a number of simple sentences in a 75 sentence 
sample (x^). A regression equation was derived for determining the 
difficulty of those books reported as being read and liked. The equa- 
tion for predicting grade levels as measured by the Standard Para- 
graph Meaning Test (Y) is; 

Y = .085x1 +. 101X2 + .604x3 - .411x4 + 17.43 

A drawback inherent in this study is that the reading material sample 
included only those books which were read and liked. Accordingly, 
the sample is confounded by the evaluation of the books by the students. 
However, the Washburne and Vogel technique is significant historically 
becciuse it: (1) analyzed the effects of structural factors in the text, (2) 
employed an objective criterion (textual difficulty) instead of qualitative 
iudgment, and (3) described readability in terms of -'-hool grade levels 
rather than in terms of a relative measure. 

The two readability techniques, discussed previously, that of Lively 
and Pressey and that of Washburne and Vogel, are typical of the early 
measures which emphasized readability measureme»it as u function of 
vocabulary uniqueness with reference to Thorndike's word list, and 
which used relatively crude criteria of the difficulty of the text. 
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studios During the 1930s 



The readability studios undertaken during the 1930s wore largely 
characterized by: (1) emphasis on evaluation of a greater number of 
readability factors, (2) reduced emphasis on word frequency lists (e.g., 
Thorndike's 10,000 most commonly occurring words in the English lan- 
uage), and (3) a trend to nonanalytic methods, i, e, , concern over more 
objective readability criteria such as comprehension test scores. 

For example, Ojomann studied the use of comprehension tost 
scoi'es as the criterion of readability. Ho administoi'od 10 passages, 
each of approximately 500 words, to adults and then tested their com- 
prehension of each passage. The passages were arranged in order of 
difficulty and their contents were analyzed for eight sentence factors, 
six vocabulary factors, and three qualitative factors. Instead of a formu- 
la for determining readability, Ojomann presented his 16 listed selections, 
along with the values he determined for the quantitative aspects (i, e, , the 
vocabulary and sentence factors), and their respective difficulty levels. 
This type of presentation was intended to allow other analysts to evaluate 
the qualitative factors in now passages, in addition to comparing the new 
passages with his tested passages on the quantitative factors. While 
such a technique is cumbersome and time cons\iming, Ojemann's study 
is noteworthy because it was the first to; employ adult subjects, use com- 
prehension test s joros as a criterion of readability, and demonstrate the 
value of qualitative, nonstatistical factors in determining readability. 

However, this period was not devoid of some continued emphasis 
on the devolopment of formulas. For example, Dale and Tyler asked 
adults of low reading ability (third to fifth grade level) to take a compre- 
hension test after reading health education articles that i.pMoared in news- 
papers, magazines, and books. Adequate tost scores wor*. not obtained. 
Dale and Tyler rewrote the passages, making them much simpler to read. 
They found that using very basic vocabulary, an informal style character- 
ized Dy a convei'sational manner and anecdotes, and nevoi' digressing 
from the main topic increased the rtKidability of the articles. Their anal- 
ysis, based on the roadministration of the r-nvritten passages, resulted 
in several factors correlating significantly w:l)i comprehension. Three 
of the factors were included in a regression equation for predicting com- 
prehension: the number of technical words (x|), the number of nontech- 
nical words not known to 90 per c(;nt of sixth gr;ide pupils, as determined 
in a study performed by Dalc^ (X2), anu the numb(^r of indeterminate 
clauses (X3), The equation used to predict comprehension (Y) is; 
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Y = -9. 4x^ - 0. 4x,, + 2. + 1 14. 4 

One study which fully exemplifies the trend in formula exten- 
sion during this period is that of Gray and Leary. They listed over 
80 factors related to readability. Through various processes of elim- 
ination. Gray and Leary were able to develop regression equations 
based on only a few factors. The most frequently used of their formu- 
las was one consisting of five factors. This formula is 

y = .01029X., + .009012X. - . 02094x^ - .03313x_ 
I o U I 

-.01485Xo + 3.774 

o 

where: 

Y = predicted average comprehension score, along a 

scale of + 4 to -4 

X = average number of difficult words (words not ap- 
pearing on Dale's list of 769 easy words) 

x-= average number first, second, and third person 
pronouns 

Xg= average sentence length in words 
x^= average percentage of different words 
x_= average number oi prepositional phrases 

o 

Since this formula is most applicable to low ability readers, it lacks b 
utility. 
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studies of the 1940s and Early 1950s 



The trend in readability research during the 1940-1950 decade 
tended toward more efficient and more easily applied formulas, Lorge 
modified the formulas developed by Gray and Leary and obtained higher 
correlations with his criterion <:f reading passages than did Gray and 
Leary* Lorge attributed this increased correlation to his use of a great- 
er and more adequatelj'^ standardized set of criterion materials, Lorge's 
criterion materials were the Standard Test Lessons in Reading of McCall 
and Crabbs, The Standard Test Lessons in Reading consist of 376 read- 
ing passages containing norms related to grade levels. The formula 
Lorge used included two of Gray and Leary's factors, Xg (average sen- 
tence length in words) aud xg (number of prepositional phrases per hun- 
dred words), in addition to the factor xg (ratio of hard words to total num- 
ber of words, hard words being those that do not appear on Dale's list of 
769 easy words). The final readability formula was: 

= . lOx^ + , 06Xg + . lOXg + 1, 99 

where: 

CgQ = reading grade level of individuals answering 

50 per cent of the McCall and Crabbs questions 
correctly 

Flesch was dissatisfied with the then current readability formulas 
because of the poor relationship found between the rankings of magazine 
articles (as determined by the formulas) and the rankings of the educa- 
tional levels of the magazines' readers. He thought that the misranking 
was due to an emphasis on vocabulary and insufficient stress on other 
factors. He suggested three factors that should highly influence readabil- 
ity for adults: (D sentence length, (2) abstracts, as determined through 
the number of affixes (i, e, , prefixes and ^suffixes), and (3) number of per- 
sonal reference words (i, e, , pronouns., names, and words relating to 
people such as aunt, baby, etc.) Flesch's initial formula underwent a 
revision several years after it was first developed. This revised formu- 
lation which yields a reading ease and a human interest index has become 
widely used for readability analytic purposes. This revised formulation 
for measuring readability will be discussed in another section of this chap- 
ter. 



12 



Dale and Chall employed Flesch's original formula to evaluate 
the readability (for the average adult) of educational material published 
by the National Tuberculosis Association. They sought to develop their 
own measure of readability because they were dissatisfied with two as- 
pects of the Flesch formula. To them, counting of affixes (prefixes 
and suffixes) was not completely manageable. Dale and Chall found that 
different ratei;s, counting affixes in the same passage, came up with dif- 
ferent numbers. However, they did agree with Flesch that a measure 
of abstractness was of value and had a place in a readability formula. 
Dale and Chall's second objection to the Flesch formula related to the 
use of personal words. They felt that Flesch's interest formula was 
overly simplified and that the definition of personal words (for purposes 
of readability) should be more restrictive. Dale and Chall's formula 
for measuring readability will also be discussed more fully in a subse- 
quent section of this chapter, since it is also a popular formula for 
measuring readability today. 

Another readability formula produced in this era was that of 
Powers, Sumner, and Kearl. Using the revised passages of McCall 
and Crabb's Standard Test Lessons in Read-.n^ (which was updated so 
that 60 passages dealt with then modern topics). Powers, Sumner, and 
Kearl recomputed Flesch's formulas, as well as others that had been 
developed earlier. They intended to produce formulas which would ac- 
count for changes in reading abilities over the period between the two 
test editions of the Standard Test Lessons in Reading (some 24 years) 
and to facilitate comparisons of the formulas they recomputed. The re- 
searchers claimed that the revised formulas provided estimates more 
consistent with one another than did the original Flesch and Dale-Chall 
formulas. 

McElroy's entry into the field of readability resulted in a meas- 
ure referred to as the "fog count. " The fog count is also related to the 
Flesch formulas in that it too counts syllables. Weights are assigned to 
words such that one and two syllable words receive a value of 1; a value 
of 3 is given to each remaining word. The sum of these weights equals 
the fog count. The reading grade level was determined from the fog 
count in the following manner: divide by 2 for sums of 20 and over; sub- 
tract 2 from sums under 20, and then divide by 2. 
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Recent Studies 



The trends continued in the 1940 -early 1950 period. However, 
there has recently been some emphasis on linguistic theoretic ap- 
proaches to comprehensibility. Spache, in 1953, developed a formu- 
la to predict p imary grade level (grades 1-3) of textual material on 
the basis of sentence length and percentage of words not appearing on 
the Dale list of 769 common words. The formula is: 

Grade level = . 141x^ + . OSSx^ + . 839 

where: 

average sentence length in words 

percentage of words not appearing on 
Dale'F list of 769 common Words 

Wheeler and Smith, in 1954, developed a formula which equated 
grade level to 10 times the product of the mean length of units (sentences) 
in words and the percentage of multisyllable words. The value deter- 
mined by the multiplication of these factors was located in a table and 
grade levels 1 through 4 could then be read from the table. 

Tribe, in 1956, used the revised form of the McCall and Crabbs 
Standard Test Lessons xn Reading as the criterion of readability. The 
grade level score of pupils, in grades 2 through 8, who could answer 50 
per cent of the reading test questions correctly is: 

.0719x^ + . 1043xg f 2.9347 

where: 

- average sentence length in words 

x_ = percentage (times 100) of words not 
appearing on a standard word list 
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In 1954 and 1958, Flesch introduced three alternate readability 
indices These have not become highly popular, possibly because all 
of the indices are based on measures of word counts. The counted 
values are converted to arbitrary scales by referring to conversion 
tables. Flesch' s "r" score is an index of realism, based on the num- 
ber of references to specific human beings, their attributes or posses- 
sions, locations, objects numbered or named, dates, times, and col- 
ors. His "e" score is an index of energy, based on indications of voice 
communications, such as inflection. Finally, the "formality -popularity" 
scale is based on the total number of capitalized, underlined or itali- 
cized words, numbers (not spelled out), punctuation marks, symbols 
(§, $), beginnings of paragraphs, and endings of paragraphs. 

Coleman, in 1965, developed four formulas using from one to 
four variables to predict readability. The readability criterion was 
the mean "cloze" score on the passage achieved by a sample of college 
students. The "cloze" score is the percentage of deleted words in a 
passage filled in correctly by a reader. The "cloze" technique will be 
discussed more fully below. The simplest of Coleman's formulas, 
which correlates very highly with the involved formulas, is; 

Predicted cloze score = 1. 29 (percentage of one 

syllable words) - 38.45 

McLaughlin, In 1969, found that the number of syllables in 100 
words is equal to three times the number of words of over two syllables, 
plus 112. A very close approximation to his formula, referred to as the 
SMOG grade, is; 

SMOG Grade = 3 + square root of the polysyllable 
count in 30 sentences 

where: 

polysyllable = number of words with three'or more 
syllables 
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A formula for measuring readability v/as developed at HUMRRO 
in 1972. The measure is referred to as B'ORCAST and was dt:veloped 
for measuring the readability of Army technical literature. The devel- 
opers of this formula considered existing formulas inappropriate for 
Army technical material because extant formulas were most often based 
on school students and on general text material. Cloze scores were used 
as the criterion of readability. It was noted that cloze consistently yield- 
ed very high correlations with multiple choice tests and other more sub- 
jective measures of comprehension and difficulty. Prior research indi- 
cated that if a cloze score of 35 per cent was achieved by a reader for a 
given passage, then he might be expected to achieve a score of 70 per 
cent on a multiple-choice test of the same reading passage. Hence, 
cloze scores were demonstrated to be good indicators of comprehension 
and reading achievement level. The FORCAST formula for predicting 
reading grade level (Y) is: 

Y = 20 - (number of one syllable words oc- 
curring in a 150 word sample/ 10) 

Smith and Senter's readability analytic technique was unique be- 
cause it was based on data which could be collected from mechanical 
counters that are installed on an IBM Selectric typewriter. The tech- 
nique allows measurement of readability at rough draft typing speed. 
The mechanical counters record the number of key strokes, spaces, 
and sentences. The mean number of words per sentence (number of 
spaces divided by the number of sentences, w/ s) and the mean length 
of words (number of strokes divided by the number of spaces, s/w) may 
be computed. The regression equation which predicts grade level (GL) 
from the above ratios and which was based on graded school texts is: 

GL = 0.50 (w/s) + 4. 71 (s/w) - 21.43 

This formula may be simplified to: 

GL = w/s + 9(s/w) 

As such, it is refcx^rod to as the Automated Readability Index (ARI). In 
addition to the speed of data collection that it affords, the advantages of 
the AHI technique are: (1) its objectivity, (2) its reliability, and (3) the 
ease with which it may be incoi-porated into modern computerized type- 
setting machinei-y. 
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The Flesch Method 



The most widely employed Flesch formulation rests on two in- 
dices-dreading ease and human interest. The reading ease (RE) for- 
mula is: 

RE = 206. 835 - . 846 WL - 1. 015 SL 

or: 

RE = 206, 835 - [1.015(SL)+ • 846(WL)1 



where: 



WL = word length measured as syllables per 
100 words 



SL = average sentence length in words for a 
given sample 



The human interest (HI) index is determined from the formula: . 



HI = 3.635 PW + .314 PS 



where: 



PW = average percentage of personal words 

PS = percentage of sentences addressed to the 
reader (including exclamations and gram- 
matically incomplete sentences whose mean- 
ing is interpreted from the context of the 
passages) 



This index tests the convert>ational style of the pabsage. The conjecture 
behind the human interest factor is that prose written in a conversational 
style will be more readable than that which is noixconversationally oriented. 

These Flesch formulas provide index v.dues which range from 0 
to 100, and have become the most widely applied of any which have been 
developed. The wide use of the formulas is due, in part, to their ease 
of application. 
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A simplification of the Flesch reading ease formula was intro- 
duced by Farr, Jenkins, and Patterson in 195 U They suggested that 
Flesch's formula use the number of one syllable words, thereby allowing 
a more rapid test, instead of counting all syllables per 100 words. Such 
an approach was found to correlate ♦ 95 with the scores produced by the 
Flesch formula. Consequently, Farr, Jenkins, and Patterson suggested 
that their reading ease formula be substituted for Flesch's. Their formu 
la is: 

RE = 1.599 (number of one syllable words) 

- 1. 015 (mean sentence length in words) - 3U517 



The Cloze Procedure 

The cloze procedure, introduced by Taylor in 1953, was demon- 
strated to rank standard reading passages in the same order as did the 
Flesch and the bale-Chall formulas* In the cloze procedure, readers 
are presented with samples of text, from vhich some words are deleted 
and replaced by blank spaces. The readers are requested to fill in the 
blank spaces wi'li the words they think were deleted. To the extent that 
the author uses the words that the reader expects and understands, the 
reader will fill in the correct words. The technique assumes that read- 
ability is a direct function of the number of omitted words which the 
reader is able to fill in. 

Taylor indicated that the ordering of cloze scores is maintained 
regardless of the system employed in word deletion. He used four dif- 
ferent deletion systems on three standard passages, each of a different 
difficulty level, and found that the rankings were the same despite the 
deletion system employed. The four deletion systems involved deleting 
every fifth word, every seventh word, every tenth word, and 10 per cent 
of the words at random. Others have reported that deleting 20 per cent of 
a passage will yield sensitive measures. 

The cloze method is free from many of the disadvantages of the 
traditional readability measures. It can be applied more appropriately to 
highly technical and unusual materials. Very technical material might be 
rated as difficult by the Flesch and Daie-Chall formulas, but not by the 
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cloze technique, if the subjects reading the passage were trained in the 
subject matter area. Conversely, the readability of the writings of 
authors such as Gertrude Stein, who write in short sentences with rela- 
tively simple vocabulary, but whose style is such that the material is 
difficult might not be accurately reflected by the Flesch and Dale-Chall 
techniques. The cloze test might be expected to reflect accurately the 
difficulty of such reading passages, 

Taylor reported correlations of . 70 and . 80 between cloze scores 
and comprehension scores received by Air Force trainees on Air Force 
technical material. A study performed by Bormuth indicated that a 30 
per cent cloze score was associated with a 75 per cent comprehension test 
score; 50 per cent cloze score was associated with 90 per cent compre- 
hension score. 

In a study performed at Applied Psychological Services, a series 
of norms was developed for use in evaluating the readability of submarine 
sonar manuals through the cloze technique. In order to obtain normative 
data, the researchers administered a representative sample of submarine 
sonar manuals, in accordance with the cloze procedure, to a representa- 
tive sample of sonar technicians. The cloze scores achieved by the sonar 
technicians constitute the normative standard against which scores achiev- 
ed by similar groups on other manuals may be compared. Thus, the 
norms allow an answer to the question, "Relative to the group of sonar 
technicians in the normative sample, what percentage of technicians can 
be anticipated to achieve a given comprehensibility score on this new man- 
ual- 

The inherent advantages of the cloze procedure are: (1) scoring 
ease, (2) scoring reliability, (3) ease of application to nonstandard materi- 
al, and (4) accounting for the reader's interest in and prior knowledge of the 
content. The disadvantages of the procedure are: (1) cloze is a measure 
of readability, not a predictor of readability, (2) a sizeable sample of 
subjects is required, and (3) it may not reflect all types of comprehen- 
sion. 
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The Dale and Chall Method 



The readability formula developed by Dale and Chall was based 
on the average sentence length, and the ''Dale score. '' The "Dale score'* 
is the relative number of words in the passage that do not appear on a 
particular list of words, a list of 3,000 words known to 80 per cent of 
a sample of fourth graders. The Dale and Chall readability formula is: 

X = . 1579 (Dale score) + .0496 (sentence length) 
""^^ + 3.5365 

where: 

X^gQ = the reading grade level of an individual able 
to answer 50 per cent of the comprehension 
test questions correctly 

Reading Level Determination 

Several of the readability formulas discussed above predict read- 
ability in terms of reading grade level (RGL). RGL may be determined 
through several different approaches. The formulas above predict RGL 
as measured by standard tests of reading ability, for example, McCall 
and Crabbs' Standard Test Lessons in Reading, These tests may be used 
for groups or individuals. An annotated list of these tests is available in 
The Sixth Mental Measurements Yearbook (Buros, 1965). 

Although the standardized tests of reading comprehension have 
demonstrated validity and reliability, there are more efficient techniques 
available for use in the military situation. Madden and Tupes provided a 
method for estimating RGL from Air F'orce aptitude test scores. They 
found that the general aptitude index (AI) of the Airman Qualifying Exam 
(AQE) correlated .70 with RGL. This high correlation was largely attri- 
buted to the reading vocabulary subtest in the AQE. In predicting the 
RGL of an individual Airman, one of the following three formulas may be 
employed, depending upon the Airman^s career field: 
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Administrative: RGL = .0437 (Gen AI) + .0501 (Ad Al) + 5.0730 
Mechanical: RGL = . 0991 (Gen AI) + . 0085 (Mech Al) + 5. 0459 
Electronic: RGL = . 0743 (Gen AI) + . 0222 (El AI) + 4. 6088 

Another regression equation for predicting an individual's RGL, 
in the military, was developed by Caylor, Sticht, Fox, and Ford (1972). 
Their formula uses the individual's Armed Forces Qualification Test 
(AFQT) score. The formula is: 



RGL = .75 (AFQT score) + 5.52 
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II.. APPLICATIONS FOR INCREASING READABILITY 



Commiinication through writing, in a learning situation, is dif- 
ficult because of the many complexities involved. Often the language 
that must be used is technical, students' reading skills arc low, and 
the subject matter material is difficult to present in an interesting 
fashion. Despite the difficulties inherent in transferring information 
through the written word, textual presentation is a communication tech- 
nique that trainers must use. Written material is considered maximal- 
ly effective when a student can gain a clear understanding of the content 
after a single reading. Accordingly, it is imperative for the writer to 
use language that will be meaningful to the student— language that is 
easily understood. 

This section suggests methods for increasing the readability of 
material prepared for military training purposes. Because multimodal 
presentation of the material to be learned has been recently suggested 
in a number of contexts, this concept will also be discussed. Although 
blanket statements about the benefits that can be gained from a multi- 
modal presentation of material cannot be made for all ty^es of material 
or for all types of learners, consideration should be given to multimodal 
presentations when: (1) the material lends itself to presentation in more 
than one mode, (2) the increased benefits outweigh the increased costs 
(section IV), and (3) readirig grade levels of personnel to be trained are 
low. 

Gains to be Anticipated 

The gains that can accrue from an effective presentation of train- 
ing material are; 

• less time to achieve the required skills and 
knowledges 

• less effort required to learn the material 

These factors translate directly to increased effectiveness 
through lower costs both in the training center (through the savings in 
time to train the student) and in the field (through the diminished amount 
of time required for on-the-job training). 



There uro certain types of material that ai'e more easily grasped 
and retained when they arc presented in writing. Some types of mr.teri- 
al require time to be absorbed. The time needed to absorb the material 
can be taken by the student if he roads the material, but he may not have 
the time he needs if he is unable to control the rate of presentation. Por 
example, the student must pace himself according to the speed set by the 
instructor in the lecture, whereas he can move at his own pace when read- 
ing written matei'ial. 

Logical and Precise Writing 

Logical and precise writing is the key to improving readability, 
Quito obviously, readability will be improved if logic and precision are 
considered in the context of the structural factors suggested by the vari- 
ous readability formulas. 

When preparing readable material, it is important to know the 
reading ability of those for whom the material is being prepared. Cer- 
tainly, a writer of textual material would not write at an eigth grade 
reading level if he were preparing course material for college seniors. 
On the other hand, it would be folly to write a discourse at the sixteenth 
grade reading level when it is to be read by enlisted Airmen taking a tech- 
nical training course. Therefore, any "rules" for preparing written ma- 
terial must be tempered by the background and reading grade levels of the 
readers, 

Flesch suggests: 

1, preparing the material in a personalized manner— in- 
crease the number of personal words (e,g, , I, you, he, 
him, she, her; words of masculine and femipii).e gender 
such as John, Miss, mother, brother; and group words 
such as people) and increase the number of personal sen- 
tences (e. g. , spoken sentences in which quotation marks 
arc used; sentences addressed to the reader as in 'do 
this" and "don't do that;" exclamations; and incomplete 
sentences) 
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2. discussing all points presented 



3. shortening the length of words and sentences (if 
long technical words have to be used, make sure 
the meaning will be clear to the reader even if a 
definition must be included) 

4. using punctuation as an aid to the reader 

5. discussing points in chronological order or in order 
of increasing difficulty 

6. avoiding excessive wordiness 

7. providing anecdotes or illustrative examples wherever 
possible 

8. saving the most important points for last; many learn- 
ers remember best what is presented last 



Learning from textual material will also increase when the follow- 
ing are included: 

1. an introduction outlining the points that follow 

2. a summary at the end 

3. major and minor headings to guide the reader 

4. transitional paragraphs which lead into subsequent 
topics 

5. statements telling the reader what the important points 
are— telling him what to remember 



25 



In addition to those aspects outlined above, which all serve to 
improve clarity, the writer should present one point at a time and stick 
to that point. Deviations from the main theme must be avoided and im- 
portant points within the theme must be emphasized. Emphasis can be 
accomplished in several different ways; e. g. , underlining, italics, repe 
tition, a statement such as "this is an important point, " and devoting 
more space to the points which are to receive greater emphasis, 

A conversational writing style also enhances readability because 
one tends to present his ideas simply when speaking, but may become 
a little more complex when committing his thoughts on paper. As an ex- 
ample, an instructor might say to his class "Tomorrow you will have a 
10 minute multiple choice test on Chapter 2. " The message is clear and 
concise. If a literary written instruction was given, it might be worded, 
"In order to determine your level of achievement on the subject of bio- 
logical warfare, it is recommended that you be examined at the comple- 
tion of the subject. Therefore, a paper and pencil test will be admin- 
istered. " The consequences of such an instruction could be sizeable 
to the instructor (if the students require clarification from him person- 
ally) or to the students (if they appeared at the next session of the class 
unprepared for the test). Although written instructions may be better 
than oral instructions because they are available for additional refer- 
ence, they should be direct, concise, and follow the style of the spoken 
version in order to be effective. 

Other suggestions which will assist the writer in preparing more 
effective written materials are: 

1. Do not begin a sentence with the indefinite pronoun 
"this. " "This" refers to something in the preceding 
sentence or paragi aph, but the specific antecedent 
may not be clear to ilie reader. Since the goal is to 
increase comprohensibility, repeat a word or concept 
rather than chance being unclear to the reader. 

2. Avoid using colloquial expression which may be mean- 
ingful to certain groups of people but not to others; e.g., 
"groovy, " instead of "very acceptable. " 
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3. Use parentheses to set off explanations. Do not 
enclose one set of parentheses by another; use 
brackets to enclose parentheses; e.g., 10[(a+b) 
(c+d)]. 

4. Use abbreviations only after a full explanation 

of the abbreviation has been provided, except where 
the abbreviation is very widely known; e.g. , 
12 a. m. An abbreviation is often defined in paren- 
theses after the first use. The abbreviation may be 
used alone thereafter. 

5. Use only those symbols that are very common; other- 
wise spell out the symbol. Examples of some com- 
mon symbols are: $, ^, #, =, etc. If less popular 
symbols are needed, define them for the reader. 

6. Do not omit an article for the sake of brevity; e.g. , 
"The students and the instructors were invited to the 
party, " not "The students and instructors were in- 
vited to the party. " 

7. Do not omit the word that; e. g. , "Lt. Smith felt that 
he should speak to Pvt. Jones, " not "Lt. Smith felt 
he should speak to Pvt. Jones. " 

8. Do not omit a verb; e. g. , "He finished the course on 
June 6 and was tested on June 8, " not "He finished the 
course on June 6 and tested on June 8. " 

9. Place modifiers as close to the word thej' modify as 
possible; e. g. , "He studied diligently for three hours, 
not "He studied for three hours diligently. " 

10. Avoid splitting infinitives unless the sentence would be 
awkward; e.g., "She wanted to return promptly after 
Christmas, " not "She wanted to promptly return after 
Christmas. " 
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11, Avoid using the wox^d and to connect two unrelated 
ideas; either rephrase the sentence using other con- 
nectives or write two sentences. For example, ''Since 
the instructor taught the material, the students were 
able to answer the question,'' or, "The instructor taught 
the material. Thus, the students were able to answer 
the question;" but not "The instructor taught the materi- 
al and the students were able to answer the question, " 

12. Use the active voice rather than the passive voice. For 
example, "Use a screwdriver to help you raise the lid;" 
not "a screwdriver should be used to help raise the 



Word and Sentence Simplification 

The selection of words is crucial to preparing readable copy, A 
general principle to follow is to select words that have a high frequency 
of occurrence and are thereby familiar to the readers. In addition to se- 
lecting familiar words, the words used should be: 

a# short 

b. nontechnical (where possible) 

c. American sounding rather than foreign sounding 

d. concrete rather than abstract 

The next logical consideration is sentence structure. Here, the 
basic factor is the length of the sentences. The characteristics of the 
reader will be the largest single factor in determining how long or short 
sentences should be. Generally, shorter sentences are more readable. 
However, other sentence structure characteristics should be considered 
in making written material more readable. The characteristics to be 
avoided are: 

1, prepositional phrases (e.g.. The two roads of 
which I speak are. , . ) 

2, compound sentences (e.g, , He campaigned for 
nomination, ran for election, and won the vote, ) 

3, complex sentences (c;*g. , I would have passed 
the tost if I studied more. ) 
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Since the more complicated sentences are usually longer than 
simple sentences, it can be demonstrated that sentence length is direct- 
ly related to sentence structure. Considering that sentence length is 
easier to analyze than sentence structure, length is the more frequent- 
ly used factor in readability measurement. 

The following examples represent excerpts from Air Force 
manuals, as they appeared in their original form and then in a modified 
version.. The modified versions demonstrate the effects word and sen- 
tence simplification. 

Original: The Air Force uses two basic methods— prepost and post- 
post— for issuing supplies. A prepost issue is processed 
through the computer before the property is actually se- 
lected from storage. A post-post issue made before the 
issue request is processed through or into the computer, 
A simple way to determine which method is being used 
is that prepost documents are machine printed and post- 
post documents are normally handscribed. 

Modified: The Air Force uses two methods for issuing supplies. 
These are prepost and post-post, A prepost issue is 
processed by the computer before the supplies are taken 
from storage, A post-post issue is processed by the 
computer after the supplies are taken from storage. You 
can easily tell which method is being used because pre- 
post order documents are printed by machine. Post- 
post order documents are usually written by hand. 



Original: Post-post issues from the base service store are ac- 
complished in the same manner as normal warehouse 
post -post issues. The AF Form 2005 and the DD Form 
1348-1 are handscribed. Unlike the single item pre- 
post issue, the property is not removed from stock un- 
til the document arrives at the BSS. If the entire quan- 
tity requested is not available, the quantity actually is- 
sued is entered in the quantity block of the DD Form 
1348-1, The property and documents are forwarded to 
the pickup and delivery section for delivery and docu- 
ment distribution. 
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Modified: Post-post issues from the base service store are 

made the same way as regular warehouse post-post 
issues. The AF B'orm 2005 and the DD Form 1348-1 
are filled in. The procedure is different than that 
used for a single item prepost issue. Supplies are 
not removed from stock until the issue form arrives 
at the BSS. If you don't have enough of an item in 
stock to fill an order completely, write the amount 
that you can supply in the quantity block of DD Form 
1348-1. The items and forms are sent to the pickup 
and delivery section. These people will deliver the 
items and distribute the forms. 



Original: Copy 1 of the AF Form 1991 for a location addition 
is forwarded to the warehouse remote operator for 
input, or to the punch card accounting machine (PCAM) 
unit for keypunching and input via the card reader. 
Copy 2 is held in a suspense file, pending receipt of 
the output notice or its appearance on the daily docu- 
ment register. If the input is by a remote device, an 
output notice is produced on the retnote. If the input 
is made by the card reader, no printed notice is pro- 
duced. 

Modified: If the new warehouse location input to the computer 
will be through remote, send copy 1 of the AF Form 
1991 to the warehouse remote operator. An output 
notice will be produced on the remote. If the input 
will be through a card reader, send the form to the 
punch cjird accounting machine (PCAM) operator. He 
will keypunch and enter the information. No output 
notice will be produced in this case. Keep copy 2 of 
the AF Form 1991 in a suspense file. When you re- 
ceive the output notice, remove it from the file. You 
also remove copy 2 from the suspense file if that lo- 
cation input appears on the daily document register. 
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Repetition -Redundancy 



Repetition is an important concept in readability-comprehensi- 
bility. Repetition reduces uncertainty in the reader by helping him to 
remember impor<-ant points. 

To increase redundancy, it is not necessary to repeat specific 
points verbatim or even to repeat the same point using different words. 
In some cases this could lead to monotony; those who wish to reread a' 
specific section could do so. Increasing redundancy can be accom- 
plished by writing about a specific issue from another point of view. Re- 
dundancy can also be increased by using words in a sequence; o. g. , he 
is loud, dumb, and vulgar. 



Textual Supplementation 

Reading is a complex process involving the reader's education, 
reading techniques, intellect, and emotional status, as well as the char- 
acteristics of Xhe textual material itself. Other factors affecting the abil- 
ity of the reader to read quickly and understand textual material are the. 
reader's visual acuity, his physical condition, his educational background, 
cultural factors, distractions present, the reader's ability to conceptual-' 
ize language, to think symbolically, and to organize the material mean- 
ingfully. Diagnosis of a reading disability requires specially trained per- 
sons. A student's inability to master the wr-itten word may be overcome, 
to some degree, by supplementing the textual material with auditory tape's 
pictures, graphs, and the like. 

Auditory Supplementat ion 

Although listening has an advantage over reading because it con- 
tains intonational nuances, individual differences among listeners exist 
insofar as rate of listening comprehension is concerned. Listening to 
tape recordings is somewhat like listening to a lecture, except for the 
formality of the classroom environment. However, the student can stop 
the tape at any time to think over a point. And, he can return the tape 
at an eai-lier point in case he has missed the point. Most research in- 
dicates thai no student will do worse with taped material than with written 
texts and that some students will do better with taped materials as com- 
pared to written texts. 
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In preparing the material for a recording, the problem of making 
the material comprehensible to the learner is as important as for writ- 
ten material. The subject matter must be efficiently organized if the 
material is to be meaningful to the listener. Learning from tape record- 
ings will be enhanced by the following: 



1. limit the amount of information on any one tape 

2. repeat key points at strategic places in the recording 

3. keep the material at the level of comprehension of 
the students 

4. permit discussions and questions before and after 
the student listens to the recording 

5. prepare the student for the recording by telling him 
of the subject matter to be covered, its purposes, and 
how he will be able to use the information 

6. save the most important points for presentation near- 
the end 

7. compliment the students by leaving them with a note 
of encouragement 



The speaker on a tape recording should be skilled in the techniqu 
of verbal delivery; i.e. , someone trained in the area. Since the speake] 
on a recording cannot take advantage of his body and other audio-visual 
aids, he will have to concentrate on different types of verbal techniques 
to project his ideas and purposes. Some of the key points in making a 
good delivery are; 



1. present the nr^jiterial in a lively and animated fashion 

2. convey a sense of enthusiasm and belief in the im- 
portance of the subject matter. The material must 
b(.' learned and understooc' by the speaker in order 
to deliver the talk with conviction. 

3. use a pleasant tone and vary the pitch and force. The 
rate should be caretully controlled as should the degree 
of loudness. 
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4. use pitch, tone, and rate variation for emphasis 

5, use inflections — a downward inflection at the end 
of a complete thought (as is typically used at the 
end of sentences); an upward inflection at the end 
of a question (often associated with surprise, doubt, 
uncertainty). Do not use rising inflections at the 
end of complete thought units, as it presents a sing- 
song quality 

6, use short, crisp sentences 

7. use a voice that is free from any unusual quality 
(e.g., harshness, nasality, breathiness, hoarse- 
ness, throatiness) 

Illustration Supplementation 

There are other alternative techniques available for presenting 
information that one might normally present in prose. These alterna- 
tive approaches may result in improved comprehension and/ or perform- 
ance. Although an instructor may use what he considers to be simple 
and concise language (and the writer or reading material does the same) 
communication might be less than complete without visual presentation. 
Visual presentations show the learner exactly what the writer, is talking 
about or what is being referred to in the written material. Illustrations 
are held to be essentially demonstrations but in a remote fashion. They 
save time because fewer words have to be written to describe a situation 
and, accordingly, fewer words have to be read. 

The following should be taken into consideration in preparing il- 
lustrations: 

1. avoid unnecessary artwork since it may seive 
only to distract 

2. colors, if used, should be in sharp contrast with 
each other 
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3. if temporal sequencing is involved (e. g. , as in a 
step-by-step procedure) and only that part of the 
equipment is to be visible which applies to a par- 
ticular step, then ovei'lays on transparancies 
should be used 

4. limit the number of callouts or identifiers 

5. connect identifiers to the correct point in the 
illustration with a leader line 

6. limit the callouts to those referenced to in the 
text 

7. use directional arrows to help the reader orient 
himself with respect to the illustration 

8. all text appearing on the illustration sjiould be in 
an area which will not affect the readability of the 
illustration, its callouts, or identifiers 

9. simplify as much as possible (an overcrowded il- 
lustration is confusing) 

10. use size and placement for emphasis 

11. discuss the important aspects of each illustration 
in the text and state how to apply the information 

12. keep the illustration interesting --from the reader's 
viewpoint 

13. choose a simple, easy to read letter style 

Preparing effective illustrations, a field in itself, requires pro- 
fessionals and experts. Commercial and industrial artists, engineers, 
and draftsmen are often consulted in these matters. However, since 
these professionals may not be subject matter experts, effective illus- 
trations will represent the merging of instructors, who know the subject 
matter and message to be delivered by the illustration, with profession- 
als who have the artistic talent. 
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Flowcharts 'and Tables 

Flowcharts and schematics also represent forms of textual sup- 
plementation. The flowchart is one form of illustration which is h6lp- 
. ful when several complex, serialized points have to be made or when 
the subject matter pertains to interactive sequences of acts that should 
be performed under different conditions. The flowchart is more easily 
understood, more accurately used, and more quickly interpreted than 
its prose counterpart. Figure 1 presents a set of instructions in prose 
form. 



Instructions in Prose 

Place switch S-1 on the tester in the "on" position. 
Probe connections are required from scope input A to TP3 
on the LLM. Adjust the OUTPUT LEVEL on the LLM so 
that the amplitude is 5 v. P/ P. If there is no output at TPS, 
check for a signal at VI 01, pin 1. If there is a signal at 
VI 01, pin 1, then check for a signal at V102, pin 2. If 
there is no signal at V102, pin 2, there is trouble in the 
filter network. There is trouble in either V102, TlOl, or 
associated circuitry if there is a signal at V102, pin 2. If 
there is no signal at VlOl, pin 1, check for a signal at 
VI 01, pin 2. If there is a signal at pin 2, the trouble is in 
the . . . , but if there is no signal at pin 2, then check VI 03, 
pin 3, for a signal 



Figure 1. Sample of instructions in prose. 



These same instructions are also presented as Figure 2 in the 
form of a flowchart. It. can be seen, from Figure 2, that the flowchart 
will help the reader who is searching for a specific piece of information. 
It also provides an integrated picture so that relationships and orderings 
among the parts can be readily perceived. 
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Figure 2. Example of flowchart to represent instructions of Figure 1. 
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Another alternate to prose is tabular presentation. Figure 3 is 
a two-dimensional table, presenting the same information as in Figures 
1 and 2. While complex information is difficult to present, regardless 
of the format, it is evident that for the information presented in Figures 
1, 2, and 3, the prose version is the least comprehensible. 

The advantages of presenting quantitative data in a tabular for- 
mat are that tables are: (1) economical, (2) systematic, (3) concise, and 
{■\) comparisons are facilitated within and among tables. The following 
will help in preparing comprehensible tables: 



1. title the table with a brief descriptive statement 
of the contents 



2. legend each column 

3. legend each row 



4. separate classifications (i.e., between rows and 
between columns) by ruling in lines or leaving 
wide spaces 

5. identify the units that are used for the numbers 
in the column (e. g. , dollars in thousands, etc. ) 

6. use footnotes to point out any unusual circum- 
stances or information that cannot be placed in 
the body of the table 
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Graphs 



Graphs are of particular value when the writer wishes to display 
the general shape of a function, or where trends or relationships between 
quantities are bei' g described and contrasted. There are several differ- 
ent forms of graphs. The three most popular are the line graph, the bar 
graph or histogram, and the circle or pie graph. The graph should be 
placed close to the place where it is mentioned in the text. 

In the typical line graph, when the functions portrayed are con- 
tinuous, a set of values is described on the. horizontal or x axis and 
another set of values is described on the vertical or ^ axis'. When plot- 
ting the results of an experiment, the independent variable, by conven- 
tion, is laid off on the horizontal axis and the dependent variable is laid 
off on the vertical axis. The following will help in the preparation of 
comprehensible line graphs: 

1. draw the horizontal and vertical axes with heavy 
and distinct lines 

2. label the axes in terms of the units used (e. g. , dol- 
lars in thousands, percentage, etc.) 

3. title the graphs descriptively but as concisely as 
possible 

4. place the title below the graph; never place it within 
tne grid or plotting area 

5, select convenient scales for both axes; the scale se- 
lected must rema^ the same from the beginning of 
the axis to the enu 

6, select units that are easy to divide (e. g. , 1, 2, 5, 10, 
50, 100, etc.) so that intermediate points will be rela- 
tively easy to read 

7, construct the axes so that the vertical axis is 2/3 to 
3/ 4 as long as the horizontal axis 

8, arrange the graph so that it is read from left to right 
and from the bottom to the top 
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9. assign the value of zero .o the point where the 
two axes meet; if the data are such that there is 
a large gap between zero and the next value, the 
axis is discontinued between zero and the first 
value by using the symbol « (see Figure 4) 

10. use dots or crosses to plot values corresponding 
to the intercept points on each axis 

11. if several curves of different importance are plot- 
ted in the same graph, make the principal curve 
the most prominent (e.g., use a thicker line) 

12. limit the number of functions plotted (curves on the 
graph) to four; try to avoid a cluttered appearance 

13. provide legends wherever an explanation or an iden- 
tification of different lines is required 



An example of a line graph is presented in Figure 4. * 

The second most commonly used type of graph is the bar graph. 
The bar graph is a representation of discrete numerical quantities and is 
useful for displaying differences among these numerical quantities. For 
example, if the frequency of students from various states were to be plot- 
ted on a bar graph, the states would be listed on one axis and frequencies 
or percentages on the other. There are no rules regarding the axis to use 
for the different variables, or the lengths or heights of the bars. If there 
is no logical or ordinal relationship among the discrete entities plo+ted, 
then they may be arranged from lowest to highest. An example of a bar 
graph is displayed in Figure 5. The graph should be titled. By conven- 
tion, the title is placed below the graph. Provide supplementary descrip- 
tive information thatis required for understanding the graph in a key or 
legend. 
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The circle or pie graph displays diffeionces among the data in 
terms of different size sectors or segments. It is useful for demon- 
strating the relationship among parts of a whole. It is a simple graph 
to draw, and easy to understand. To prepare an effective circle or pie 
graph: 

1. title the graph—below the circle 

2. legend each sector 

3. include in each sector the proportion of the total 
circle that is represented by the sector 

4. make the size of each sector directly proportional 
to its proportion of the whole circle 

5. use a protractor for drawing the circle and for 
plotting the percentages of each segment 

An example of a circle graph is presented in Figure 6. 



Schematic Diagrams 

Schematics have been broadly used for equipment maintenance 
instruction and on-the-job maintenance support in both military and 
commercial settings. The IMM (Integrated Maintenance Manual) and the 
SIMM (Symbolic Integrated Maintenance Manual) by the Navy and Coast 
Guard are examples of military manuals that have incorporated the use 
of concise, consistent, and well-laid out schematics of equipment. PIMO 
(Presentation of Information for Maintenance and Operation) is the Air 
Force version for presenting technical order type instructions and in- 
formation. The information in PIMO emphasizes how to do something, 
rather than what should be done. 
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EXAMPLE OF PIE GRAPH 




Figure 6. Percentage of United States' population completing 
various years of schooling, 1966. 
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Comprehensible schematics will include: 

1. easy to understand layouts 

2. clearly demarked circuits and components 

3. maintenance data 

4. the same functional divisions as the equipment 

5. aid for locating components (e. g. , grid underlay) 

6. cross -sectional schematic views of mechanical 
devices 

7. coding (e. g. , flow arrows to show direction of 
flow) 

8. wiring and cabling interconnections 

When text is included with the schematic, it should be: 

1. placed on the page facing the schematic 

2. readily associated with the schematic portion 
to which it pertains 

3. short but complete 

Figure 7 presents a schematic that incorporates many of 
principles outlined above. 
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III. "NONANALYTIC" METHODS FOR EVALUATING 
TEXTUAL METHODS 



While use of the word counting (e. g. , the Flesch or Dall-Chall 
technique) or associated (e. g. , cloze) methods will be useful in evaluat- 
ing the readability of textual materials, there are a number of nonanal- 
ytic techniques which will also be helpful to the material developer who 
wishes to evaluate his product. These nonanalytic methods are perform- 
ance oriented. They consider the actual performance of the person who 
reads the material rather than the characteristics of the text itself. Gen- 
erally, the nonanalytic techniques involve administering the written ma- 
terials to a group and, then, testing the group to determine how much 
they have learned, o.' how much they comprehended about the subject 
matter from reading the materials. 

It might be taken for granted that modifying the textual materials 
to produce a lower (and presumably easier) reading level will necessar- 
ily produce learning benefits. But it is also logical to assume that a two 
pound weight will fall twice as fast as a one pound weight. In the final 
analysis, some test of the effects of a materials modification on actual 
learning may be desirable. This chapter presents concepts and consid- 
erations important to such an endeavor. Such concepts and considera- 
tions include the design and conduct of such tests as well as the charac- 
teristics of the actual measurement instruments to be involved in the 
evaluation. 



Design of Materials Evaluations 

The evaluation of textual materials through nonanalytic methods 
will rely on the learning they produce, as measured by tests of the per- 
formance of the reader. Such tests may be administered immediately 
after the materials are read or after a time interval has elapsed. In 
the latter case, the final result will be confounded by the intervening 
activity of the reader. Thus, the intervening activity must be careful- 
ly controlled, or it will not be known whether any actual learning can 
be attributed to the materials themselves. 

Because the design and analysis of a nonanalytic evaluation is 
a Complex process, it is best to consult experts in research methods 
and statistical analysis before conducting such an evaluation. To per- 
form a nonanalytic test, study guidelines, which are specific and not 
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subject to personal interpretation, should be developed in advance. The 
guidelines must describe fully the methods, procedures, data collection 
instruments, and statistical data treatment methods to be employed: 
Estimates of the time needed to conduct each aspect of the study should 
also be included. Example methods for conducting such evaluations are 
described below. 

Single Group Methods 

Assume that the textual materials for a course have been modi- 
fied and that the materials developer wishes to know whether or not the 
modification produces any significant change in the amount of student 
learning. Assume that there are four blocks in the course involved. 
The original materials (O) could be administered to students when they 
are in blocks 1 and 3, while the modified (M) materials could be admin- 
istered to the same students when they are in blocks 2 and 4. The class 
scores on end of block 1 and 3 tests (O materials) could then be com- 
pared with their scores on block 2 and 4 tests (M materials). One would 
anticipate higher scores on the M materials than on the O materials, and 
the difference between the two could be used as an index of the gain achiev 
ed by the modification. 

In this type of study, each reauer is exposed to each type of mate- 
rial. This avoids certain problems associated with control group studies. 
However, note that this type of design assumes that learning block 1 does 
not affect learning in block 2, etc. , and that blocks 1 and 3 are at the 
same difficulty level as blocks 2 and 4. Such assumptions may not al- 
ways be appropriate, and when they are not, this method should be avoid- 
ed. 

Control Group Studies 

When the assumptions of the single group study cannot be met, 
control group studies can be employed. In the prior example, the stu- 
dents could have been split into two groups. In making such a split, it 
is essential that the two groups be equated on all variables which could 
affect the evaluative outcome. In the present context, reading grade 
level and background in the field of study might represent the critical 
variables on which the two groups would be equated. 
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There are two methods for establishing equivalent groups. One 
is to select the persons for assignment to each group from the total 
pool of available persons at random. For example, if an alphabetical 
roster is available, every other person might be assigned to one of the 
groups and the remainder would be assigned to the other group. The 
second method is to match, for the variables thought important for the 
study, as closely as possible the persons assigned to each group. Thus, 
if there is available a pool of 20 persons of known and varying reading 
grade levels and they are to be assigned to equivalent groups, the final 
matching might look as follows: * 



Reading Grade Level of Matched Subjects 
Matched Pair Group A Group B 

12. 1 
11. 8 
10. 6 
10. 0 
9. 8 
9. 7 
8. 9 
8. 7 
8. 6 
8. 0 

Having matched the groups, the control group technique involves 
administering the O materials (for all four blocks) to one group and the 
M materials (for all four blocks) to the other group and administering 
the appropriate tests. Possibly, such tests would be administered at 
the end of each block as well as at the end of all four blocks. Any dif- 
ferences between the test scores of the two groups (either on the indi- 
vidual block tests or the final examination) would then be attributable to 
the textual materials themselves. 

Factorial Design 

A still more powerful nonanalytic evaluative approach is through 
the factorial design. Using the same example, assume that the evaluator 
wishes to know whether or not there is a difference in the learning pro- 
duced by the two types of materials and also whether or not this differ- 
ence holds regardless of the general intelligence (AFQT score) of the 
student. Factorial designs permit the evaluator to simultaneously look 
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at the effects of two or more factors on learning. In the following exam- 
ple, AFQT is considered to be a factor, while difficulty of reading materi- 
als is considered to be a second factor. 
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In this example, the total group is split into two groups: high AFQT 
scorers and low AFQT scorers. Half of the high AFQT scorers receive the 
O materials and half receive the M materials. The same holds for the low 
AFQT scorers--half receive the O materials and half receive the M materi- 
als. The analysis (called analysis of variance) of the test scores of the four 
groups will tell the evaluator: (1) whether or not a difference in learning is 
produced by the two versions of the text, (2) whether or not high AFQT scor- 
ers make higher test scores than low AFQT scorers, and, most importantly, 
(3) whether or not there is a differential effect on the benefit to be gained by 
each type of material as a function of AFQT score. 

Within single group, control group, and factorially designed evalua- 
tions, there are a large number of sub approaches. Accordingly, an expert 
should probably be consulted during the formulation of nonanalytic studies to 
insure proper experimental design. 



ERIC 



Conduct of Notianalytic Studies 

No matter how powerful the design of the nonanalytic study, usable 
data will not be obtained unless most, if not all, of the following adequacy 
ditions are satisfied: 



• control- -all conditions in the situation and within the 
subjects which might affect the evaluation must be con- 
trolled; i. e. , eliminated from the evaluation conditions 
or distributed equally over all conditions 

• sample - -the readers in the experiment should be fully 
representative of those who will eventually use the ma- 
terials and a sufficient number of subjects must be in- 
cluded to allow stability in the resultant descriptive and 
inferential statistics employed 

• question realism --the question(s) asked by the study must 
be drawn from sound concepts or theory 

• assumptions --the, assumptions must be clearly stated and 
acceptable 

• objectives --the study objectives must be unambiguous and 
clearly defined 

• personnel- -the personnel who perform the study must 
possess sufficient training, experience, and sophistica- 
tion to properly conduct the experiment 

• stimuli- -the reading materials included (stimulus repre- 
sentations) must "be meaningful to or mirror the problem 
on hand, and the stimuli and stimulus combinations must 
cover an appropriate range of materials 

• criterion weighting- -if subscores are combined they must 
be appropriately weighted 

• alternate methods- -backup or alternate methods should be 
available, in case the favored approach must be abandoned 

• time and costs --the time schedule must be realistic and 
compatible \,ith available manpower; the cost estimate must 
be compatible with manpower, facility, and equipment re- 
quirements; contingency time and costs must be allowed 
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Characteristics of Useful Measure of Performance 



Aside from concise and specific methods and procedures, read- 
ability evaluations should employ performance measuring instruments 
which possess sound measurement qualities. Recommendations from an 
evaluation can be no better than the data on which the recommendations 
are based. Decisions have to be made on the basis of the readability eval- 
uative data. In order for a data set to be scientifically defensible, the 
measures on which the data set are based should meet most, if not all, 
of the following criteria: 

1. Reliability - -A measure is reliable to the extent that 
it will yield the same score when a person is meas- 
ured with it on two separate occasions. If a measure 
does not possess this attribute, it is worthless for 
evaluation purposes. There are several methods for 
assessing the reliability of an evaluative measure and 
all of them involve the calculation of a correlation co- 
efficient. Any elementary statistics textbook will in- 
struct the reader in methods for calculating and in- 
terpreting a correlation coefficient. 

2. Validity - - Validity refers to the extent that a measure 
assesses what it intends or purports to measure. It 
is possible for a measure to be highly reliable, yet 
have little or no validity. 

3. Comprehensiveness - -Any useful performance evalu- 
ative measure must sample from the total range of 
reading material and not from a part of the range. 

4. Objectivity- -Only those aspects which can be objec- 
tively evaluated should be included in the measurement 
instruments. If the evaluator's personal biases can 
enter into the data, the measurements are subjective 
rather than objective. A measure is objective to the 
extent that different persons who use the same meas- 
ure will give a student the same score. 

Differentiation - -Evaluation measures must be able to 
reflect differences in the variable(s) being measured. 
This allows decisions and comparisons to be made on 
the basis of known amounts by which one presentation 
of a written selection differs from another presentation 
of the same selection. 
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6. Relevance and Appropriateness - -Only those know- 
ledge or performance aspects which are appropriate 
for the course objectives should be included. Too of- 
ten evaluation studies focus on the measurement of 
reader reactions and not on learning or behavior on 
the job. 



Examples of Dependent Variables 



In evaluative research, the reader's response is always considered 
the dependent variable; i. e. , that which is observe^-and measured. The 
independent variable is the variable that is manipulated to see what' effect 
changes in that variable will have on the dependent variable(s). In a readabil- 
ity study, the dependent variable could ue the trainee's responses on an 
achievement test. An example of an independent variable in a readability 
study is the variation of the textual materials under test. 

In order to obtain a dependent variable measure which reflects stu- 
dent learning on the basis of reading textual material, tests will have to be 
constructed. To accomplish this, an outline of the written material should 
be prepared first. The outline need only be in terms of major headings or 
categories. The test(s) should measure a representative sample of these 
categories. Decide, in advance of constructing the test(s), on the numb« r of 
items that should be written in each area. The considerations that go into 
the selection of the number of items to include in the various areas are: the 
importance of each area in the total learning picture, the amount of space 
devoted to each area in the written material, and the objectives that have 
the greatest value. It is also known that as the number of items in a test 
increases, the reliability of the test increases. 

There are several different types cf tests that can be constructed for 
use in a readability study. Several of the most appropriate are presented 
below. For a more thorough discussion of these tests, refer to the Handbook 
for the Evaluation of Technical Training Cou rses and Students Ai^^HRL-TR- 
72-15. " ' • 
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Written Tests 

When preparing a written test (also referred to as a paper and 
pencil test), an item pool of more than the finally required number of 
items should be developed so that if replacement items are needed for 
the test, they will be available. There are several types or forms of 
test items that can be used. The advantages and disadvantages of each 
item type should be considered as well as the learning objective to which 
the item is aimed. A test item should measure the learning objective 
as directly as possible and, depending on purpose, certain item types 
are better than others. 

The multiple -choice item, along with true -false and matching 
items, are referred to as "selection", items because the student selects 
his response from alternatives that are offered to him. The selection 
items have certain advantages over "supply" items (items for which the 
student has to furnish his own response as in the essay item). The ad- 
vantages of the selection type item are: (i) test scorer personal biases 
and subjective idiosyncracies are removed from the scoring, (2) ease 
of scoring (scoring keys or machines can be used), (3) more aspects of 
the material can be tested in a given period of time, (4) with increases 
in comprehensiveness improved validity may be expected, and (5) ease 
of application of statistical item analytic procedures to resultant data. 

Criterion and Norm Referenced Testing 

If a criterion referenced test is employed to judge the readabil- 
ity oi textual materials, the effects of reading the materials are com- 
pared with an absolute standard to determine whether or not acceptable 
(criterion) performance has been achieved. The absolute standard may 
be passing a performance item which is at a given level of difficulty or 
achieving an absolute score. By contrast, the students' results on a 
norm referenced test are compared with the performance of a peer or 
reference group; e. g. , a given percentile. 

The characteristics of criterion referenced tests are that they 
indicate the degree of competence attained from reading the mfriterial 
without reference to the performance of others, and they measure stu- 
dent performance with regard to specified absolute standards (criteria) 
of performance. 
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Generally, criterion referenced tests tell us how effective the 
written materials are with regard to a specified standard of behavior. 
Criterion objectives, expressed in terms of specific behaviors, must 
be identified. Either performance testing or paper and pencil testing 
can be used with criterion referencing. In criterion referenced meas- 
urement, grades are usually S (satisfactory) or U (unsatisfactory). 

Performance Tests 

In performance tests, the adequacy of the materials being eval- 
uated is tested through the performance of the reader on tasks which 
are relevant to his present or future job. Performance tests can range 
from simulated performance through performance of job tasks using 
actual job equipment. Scoring can be based on measurement of per- 
formance in process, adherence of a final product to prescribed stand - 
ands, care and use of tools during performance, adherence to safety 
precautions, or some combination of these categories. Completion of 
a prescribed task within an allotted time is also sometimes scored. 
One of the most popular methods of scoring is through a sequential' 
checklist. In the sequential checklist, a task is broken down into the 
sequential elements which must be correctly performed if the task is 
to be completed. Each of these elements is then sequentially listed 
for scoring by an examiner while he observes the performance of the 
person being tested on the task. 

The final score is determined by summing the points. A single 
performance test can be constructed which includes items dealing with 
safety precautions, quality of the final product, and the manner in which 
the student performs. 

There are variations to performance tests which avoid the need 
for the student to perform a task. One such variation is the examinee 
recorded performance test. In this type of performance test, t^e stu- 
dent is required to write his response to a written question about a pic- 
ture or diagram of some performance aspect. His response is usually 
a checkmark or a one-word answer. For example, in one test of this 
type, the student is asked to identify an error in a picture of someone 
engaged in an activity similar to that for which the student is training. 
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The Evaluative Milieu 

It is difficult to complete an evaluative study in a training en- 
vironment which resists change. If the administration perceives re- 
vised materials as negatively affecting vested interests, many obsta- 
cles can be placed in the evaluator's path. In this case, the evaluator 
must meet with the persons who are resisting his efforts and attempt 
to persuade them to the benefits of the evaluation. Often, by making 
these persons a part of the planning committee for such a study, such 
negativity can be overcome. 
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APPENDIX A 

How to Perfom a Cloze Analysis 



The cloze procedure involves: deleting every nth word (usually 
the fifth word) in passages selected from the text under consideration, 
administering the mutilated passage to a sample of typical readers and 
asking them to fill in the blanks, and scoring the answers. 

Sampling Reading Passages 

Since i' will be very difficult and impractical to test all the read- 
ing material in a particular manual, book, or course, you will have to 
resort to a sampling technique. The sampling technique employed should 
provide samples that are representative of the reading matter and that 
provide adequate and sufficient data for analysis. To accomplish this, 
you should: 

1. Select samples of approximately 500 words, but 
not less than 250 words 

2. Select samples from each chapter or section (or 
whatever other divisions are used in the written 
material). Try to take two samples from each 
section. 

3. Use a random technique for selecting a particu- 
lar sample. A table of random numbers may be 
used (these tables are often included in standard 
statistics text books) such that a digit from the 
table would identify the paragraph to use in the 
sample. This would require prenumbering all 
the paragraphs in the reading matter. 

4. Begin each sample at the beginning of the para- 
graph. Use as much text (250-500 words) as 
necessary to provide a minimum of 50 blanks 
per sample. 
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Preparing the Sample Reading Passages • 

The words can be deleted manually or by computer program. 
The computer program would be advantageous if the samples were 
extensive in length. If alternate forms of the cloze test are needed 
(e.g. , to prevent compromising answers of two groups of subjects who 
will be tested at different times, if the same reader will be tested 
at different times but on the same passages, or if retention over suc- 
cessive time periods is of interest) different deletion patterns can be 
used This could be accomplished by deleting every fifth word starting 
with the first, second, third, or fourth word. Other deletion patterns 
such as deleting every sixth, seventh, or even tenth word could also be 
used. The five word pattern, starting with the first word in the para- 
graph is the most popular. The closer the blanks are to each other, 
the more difficult the task becomes. As the deletion pattern increases 
from every fifth word, the passage length will necessarily increase in 
order to have a passage with at least 50 blanks (e.g. , if every sixth 
word is deleted, a 300 word passage will be needed; if every tenth word 
is deleted, a 500 word passage will be needed). In applying the cloze 
technique, the following also applies; 

1. End each test passage at the end of a sentence. 
It is not imperative to use the same number of 
words in each test passage since comparisons 
will be made on a percentage correct basis. 

2. Words are defined as a unit with spaces on eith- 
er size* Thus, numbers or groups of letters 
may be considered as words. Examples of single 
words are: 100, AFB, isn't, on-the-job, and i. e. 
Punctuation marks such as hyphens, apostrophes, 
and commas are deleted along with the words. 

3. Hyphenated words are not deleted as units if the 
two parts are not bound to each other, as in on- 
the-job, but are free forms, as in readability- 
comprehensibility. In the latter case, the hyphen 
is not deleted with the word. 

4. All blanks should be of the same size. A 15 space 
blank is most commonly used. 
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Subjects 

In order to be able to generalize the results of the readability 
study, a sample of subjects must be selected that is representative 
of the population who normally read the material. Wherever possible, 
a random sample of the entire population of concern should be selected. 
This is often impossible to accomplish because the group may be sep- 
arated geographically, because of time and cost restrictions, or be- 
cause of other reasons. 

Procedures 

When administering the reading passages, a full set of directions 
should be provided to the subjects. A technique often used in this situ- 
ation is to have the subjects read the di.-ections to themselves while the 
administrator reads them aloud. The .-ections should inform the sub- 
jects of the purposes of the study, the manner in which the blanks should 
be filled in, and any restrictions in time limit (although rigid time re- 
strictions should not be used in a cloze analysis). The subjects should 
be reassured that the study they are participating in involves a test of 
the material they are reading and not of them. An example passage 
should be provided. The subjects should be given a few minutes to re- 
cord their responses in the example and then the correct answers should 
be read to them. Some other points that may be made in the directions 
are: d) enter only one word or number in each blank space, (2) guess if 
necessary since there is no penalty for guessing, and (3) write down a 
word even if the spelling is incorrect. After providing the subjects with 
an opportunity to ask questions, they should be told to begin. A sample 
set of instructions that was used in one study is presented below. 
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Instructions 



This questionnaire is made up of six reading pas- 
sages. Some of the passages come from training materials 
being used now. Others were specially written for compari- 
son. In each passage, about one -fifth of the words have been 
left out. We want you to write in each blank space the exact 
word you think is missing. 

For example, in the sentencie, "George UlcLskin^'loyi 
chopped down a dhe^ru tree," the person responding 
thought that the words Washington" and "cherry" were the 
missing words, so he wrote them in the blank spaces. 

Some of the missing words will be easy, and others 
will be quite hard. Sometimes the word will "pop" into your 
mind; at other times clues in other parts of the passage will 
help you decide on the right word. In any case, you should 
try to write in each blank the exact word you think was taken 
out. 

As you finish each section, go right on to the next one. 
Do not rush through the materials. You will have about an 
hour and a half to finish, or about 15 minutes for each section. 

In scoring responses, accept as correct those responses that are 
identical to the deleted word. Synonyms are not correct, nor are changes 
in tense. However, words are accepted as correct even if they are mis- 
spelled. 

Results 

In analyzing the results you should: 

1, Tabulate the number of correct responses on 
each passage 

2, Determine each subject's comprehension level 
by calculating the percentage of correct re- 
sponses (in the cloze procedure, with every 
fifth word being deleted, approximately 44 per 
cent correct is considered acceptable for use 
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in supervised instruction and 57 per cent 
correct for use in independent instruction) 

3. Assign a reading grade level to each passage 
(in order to do this, data on many subjects 
must be obtained and statistical procedures 
used which would allow for relating the cloze 
scores of a group of subjects to a reading grade 
level). The statistical procedures involved are 
rather sophisticated and therefore reference to 
a statistical source is recommended. Reading 
grade levels for the subjects may be obtained by 
using the Madden and Tupes procedure, as de- 
scribed in Chapter I, Reading Level Determina- 
tion. 
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APPENDIX B 

Examples of Before and After Material and Automated 
Readability Index (ARI) and Grade Level (GL) Analyses 

The following three examples demonstrated the effect on readabil- 
ity of revised reading material. The excerpts were taken from Air Force 
manuals, revised in accordance with the principles for making written copy 
more readable, and then ARI and reading grade levels were calculated. 



Before 

The receiving inchecker's first 
duty is to process inc oming ship- 
ments from the transportation sec- 
tion, commercial carriers, or par- 
cel post. He insures that the total 
number of boxes, drums, parcels, 
etc; , received are correctly re- 
flected on the receiving documents 
that accompany the shipments. He 
also checks for physical damage 
that may have occurred during trans- 
it or as the result of improper pack- 
aging or handling by the carrier. He 
explains noted discrepancies on the 
reverse side of the receiving docu- 
ment and obtains the signature of 
the carrier as acknowledgment of 
the discrepancies. Property is seg- 
regated by priority, cost category, 
and Federal supply class as it is 
unloaded. If this is not feasible, 
due to volume or time limitations, 
segregation is accomplished prior 
to processing the property through 
the receiving and inspection lines. 

Classified items are not allow - 
ed to remain on the receiving line, 
awaiting unchecking and inspection 
actions. Upon completion of receiv- 
ing checkings and inspection action, 
the receipt is input via the remote 
keyboai d printer. 



ARI = 70.50 
GL = 15.01 



After 

The most important job of the 
receiving inchecker is to process 
all incoming shipments. Shipments 
may come from the transportation 
section, commercial shippers, or 
parcel post. The inchecker makes 
sure that the number of boxes, 
drums, or other containers receiv- 
ed is shown on the papers that come 
with the shipment. He also looks to 
see if the shipment is damaged. Ship- 
ments may be damaged during shipping 
or because of rough handling or bad 
packaging. If any of the numbers are 
wrong or if he finds any damage, he 
writes it down on the back of the re- 
ceiving papers. The person who de- 
livered the material must also sign 
the bacK of the form. This proves 
that the delivery man knew that the 
shipmeit was damaged, or was too 
big or too small. While it is being 
unloaded, the shipment is separated 
by: (1) priority, (2) cost, and (3) 
Federal supply class. If the ship- 
ment is too big to operate then or if 
you don't have enough time, it is all 
right. But, it must be separated be- 
fore it goes to the receiving and in- 
spection lines. You must not make 
classified items wait on the receiv- 
ing line for inchecking and inspection. 
When the checliing and inspecting is 
done, the receipt of the shipment is 
input via the remote keyboard printer. . 

ARI = 58.79 
GL = 8.95 
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Before 



After 



Radio interference suppression 
is the elimination or minimizing of 
the electrical disturbances which in- 
terfere with radio reception, or dis- 
close the location of the vehicle to 
sensitive electrical detectors. It is 
important that vehicles with or with- 
out radios be properly suppressed to 
prevent interference with radio re- 
ception of neighboring vehicles. 

The ignition and generating sys- 
tems have been designed to suppress 
radio interference. Ignition system 
interference suppression is effected 
by a primary capacitor in the dis- 
tributor wiring harness receptacle, 
a resistor suppression in the spark 
plugs, and by shielded spark plug 
cables. Radio interference suppres- 
sion in the generating system is ef- 
fected by a choke, capacitor, filter, 
and rectifier in the generator regu- 
lator assembly. The primary capaci- 
tator is housed in the distributor wir- 
ing harness receptacle and is an inte- 
gral part of the primary circuit. 
Spark plugs are shielded individually 
by metallic braid beneath the rubber 
insulation. The generator regulator 
assembly houses the capacitor, choke, 
filter, and rectifier. 



ARI = 72,77 
GL = 16,73 



Radio interference is caused by 
electrical impulses. The interfer- 
ence can be stopped or cut down. De- 
tectors can locate vehicles. All ve- 
hicles should be checked so they will 
not interfere with the radio reception 
of other vehicles. 

Ignition and generating systems 
were designed so they won't inter- 
fere with radios. This is done through 
3 parts of the ignition system. One is 
a capacitor in the distributor wiring 
harness.. The capacitor is part of the 
primary circuit. Another is a resis- 
tor in the spark plugs. The third is 
the shelded spark plug cables. Each 
spark plug is shielded by metal braids 
under rubber insulation. There are 4 
items in the generator regulator as- 
sembly that cut radio interference, 
Thei.e are the choke, capacitor, filter, 
and rectifier. 



ARI = 56. 145 
GL = 7.73 
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Before 



After 



The process described for de- 
veloping the human component of a 
weapon system has been combined 
with evolving educational and train- 
ing methodology to form an effec- 
tive process for designing and oper- 
ating training programs- -the sys- 
tem process. An instructional sys- 
tem is an integrated combination of 
resources (students, instructors, 
materials, equipment and facili- 
ties), techniques, and procedures 
performing efficiently the functions 
required to achieve specified learn- 
ing qbjectives. Under this concept, 
we must ensure that training pro- 
grams are not bound by traditipnal 
procedures, techniques, or rou- 
tines of the day. Therefore, instruc- 
tional system designers must care- 
fully consider both current and fu- 
ture needs when identifying and de- 
signing facilities, equipment, and 
devices. This same total systems 
concept must be applied when iden- 
tifying and planning instructional ac- 
tivities. When working in this area, 
you must develop and organize learn- 
ing experiences that will enable stu- 
dents to apply what they have learned 
to new or changing situations. The in- 
structional system must employteach- 
ing techniques that will produce gradu- 
ates who will not become the victims 
of fixed habits, procedures, or atti- 
tudes, nor be bound by extreme speci- 
alization. 



A plan to design and operate 
training courses was developed. It 
is known as the system process. It 
joins the methods used to develop a 
man in a weapon system with those 
of education and training, A train- 
ing plan achieves its learning goals 
by combining a few things. They are 
resources, techniques, and proce- 
dures. By resources, we mean stu- 
dents, "teachers, materials, tools, 
and facilities. Training courses 
should not be bound by old fashioned 
procedures or techniques. Facil- 
ities, equipment, and techniques 
should be designed for current and 
future needs. Develop and plan the 
system so that students can apply 
what they learn to new situations. 
The techniques used should not make 
students creatures of habits. Also, 
students should not be bound by speci- 
alization. 



ARI = 74.783 
GL = 16.79 



ARI = 58.457 
GL' = 8.91 
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APPENDIX C 

How to Perform a Flesch Analysis 

According to Flesch, readabUity is composed of two comDonent 
parts reading ease (an estimate of the case a reader will LveTre^- 
mg and understanding what was written) and human interest <In esti^te 
of how interesting the reader wUl find the written material). ''"""'^ 

Sampling Reading Passages 

,'^^°*=f ""^ very diffi-cult and impractical to test all the read- 
Se'^^s^hftr r-^^^^^^^^^^ ^i:^^^ 

1. Select three to five samples from an article and 
25 to 30 samples from a book. 

> ,^ 

2. Samples should be taken at random. Tables of ran- 
dom numbers may be used or a scheme such that 
every fifth paragraph, or every other page, or 
every tenth page is used. Do not select introduc- 
tory paragraphs as samples (they are usually not 
typical of the style of the rest of the section). 

3. Each sample should have approximately 100 words. 

4. Start each sample at the beginning of the para- 
graph. 
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Procedures 

The following steps should be performed to determine reading 

ease: 

1. Count each word in the sample, m&rking off 
the 100th Word. 

2. Words are defined as a unit with spaces on 
either side. Thus, numbers or groups of let- 
ters may be considered as words. Examples 
of single words are: 100, AFB, isn't, on-the- 
job, and i.e. 

3. Calculate the average number of words in each 
sentence. That is, count all the sentences in 
the sample. Then, divide the number of words 
by sentences. For instance, if you have a 110 
word passage and 12 sentences, the average 
sentence length is 9 (rounding off to the nearest 
whole number). If more than one sample is se- 
lected from a piece of writing, add the total num- 
ber of sentences in all the samples and divide the 
total number of words by the total number of sen- 
tences. 

4 If you restrict the sample to exactly 100 words, 
* then the sentence in which the 100th word falls is 
counted in the sentence count if more than half 
the words in the sentence were included in the 100 
word count. For example, if the last sentence 
was "I sang for my supper, " and the 100th word 
in the sample was sang, this sentence would not 
be counted. 

5. The following punctuation marks defiiie a sentence: 
period, question mark, exclamation point, colon, 
and semicolon. However, sentence counting be- 
comes complicated with colons and semicolons in 
certain situations. In order for the, clauses to be 
counted as separate sentences, they must be in- 
dependent and a separate unit of thought. For 
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example, the following is counted as one sentence: 
The advantages of this technique are; (1) ease of 
use; (2) validity. But, the next example is counted 
as two sentences: He is tall and dark^ she is short 
and blond. 

6, Count the average word length in syllables. This 
is done by counting all syllables and dividing the 
total number of syllables by the number of words. 
In the final formula, the average word length in 
syllables is expressed as the number of syllables 
per one hundred words. Therefore, multiply the 
result of the total sample by 100, When using 100 
word syllables, count the total number of syllables 
in all the samples and divide by the number of sam- 
ples, 

7, Syllables are counted the way the word is pronounced. 
For example, average has two syllables, and read- 
ability has five syllables. Syllables, for numbers 
and symbols, are counted the way they are pro- 
nounced; e,g,, AFB has three syllables, # (number) 
has two syllables, and 34 (thirty-four) has three 
syllables, 

8, The reading ease score is determined by the follow- 
ing formula: | 

Reading Ease Score = 206. 835 - [1. 015 (average 
sentence length) + . 846 (number of syllables per 
one hundred words)] 

This result will vary between 0 (unreadable) and 
100 (easy-to read). The table below may be used 

jg^ hglp XntgrpreT iHe rfiariin^y Pag^ gf>/^y.^q^ 

Reading Ease Score Interpretation Grade Level 



90-100 Very Easy 5 

80- 89 Easy 6 

70- 79 Fairly Easy 7 

60- 69 Standard 8 and 9 

50- 59 Fairly Difficult 10 to 12 

30- 49 Difficult 13 to 16 



0-29 Very Difficult College graduate 
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To find the human interest score: 



1. Count the number of personal words per 100 
words. In testing a whole piece of writing, 
divide the total number of personal words by 
the total number of words and multiply by 100. 
If 100 word samples are used, count the per- 
sonal words in each sample and divide the tot- 
al number of personal words in all samples by 
the number of samples. 

2. Personal words are the first, second, and third 
person pronouns. The neuter pronouns it, its, 
itself are not counted. The pronouns they, them, 
their, theirs, themselves are counted if they re- 
fer to people rather than things. He, him, his, 
she, her, and hers are always counted (even if 
they refer to animals or things). 

3. Count all words that are of the masculine or femi- 
nine form, such as Betty, mother, brother, John, 
ballerina. However, words that are common to 
both sexes are not counted; e.g., lawyer, parent, 
singer, child. Doctor John Smith has one personal 
word (John); Mr. John Smith has two personal words 
(Mr. and John). 

4. Count group words; e.g. , folks and people as per- 
sonal words. 

5. Count the number of personal sentences per 100 sen- 
tences. In testing a whole piece of writing, divide 
the total number of personal sentences by the total 
number of sentences and multiply by 100. In testing 
samples, divide the number of personal sentences 

in all the samples by the number of sentences in all 
the samples and multiply by 100. 
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6. Personal sentences are spoken sentences marked 
by quotation marks or other indications that the 
sentence was spoken (spoken sentences are some- 
times set off by commas and colons). Do not 
count quoted phrases when the quotes do not in- 
dicate spoken phrases; e. g. , The Flesch i*ead- 
ability technique measures "reading ease" and 
"human interest. " 

7. Count as personal sentences all sentences that 
are addressed to the reader; e. g. , Add 3+2, 
Count all personal sentences. Do not count words 
that are common to both sexes. Do not count 
sentences that are vaguely addressed to the read- 
er; e. g. , It is a good idea to arrange the data 
neatly. 

8. Exclamations are counted as personal sentences; 
e. g. , Holy cow ! 

9. Incomplete sentences whose meaning has to be in- 
ferred are counted as personal sentences; e. g, , 
No, So, So. 

10. The human interest score is determined by the 
following formula; 

3. 635 (number of personal words per 100 words) + 

. 314 (number of personal sentences per 100 sentences) 

This result will vary between 0 (no human interest) and 
100 (much human interest). Using the following table 
to interpret the human interest score: 

Human Interest Score Interpretation 



60 to 100 
40 to 59 
20 to 39 
10 to 19 
0 to 9 



Dramatic 

Highly Interesting 

Interesting 

Mildly Interesting 

Dull 
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Example 



When thinking of "What will the student be able to do? " we ^ 
must first answer the question "What can the student already do? " 
This haa a direct impact on the analysis of each training standard 
task/ subject knowledge statement and eventually affects the depth 
and scope of the learning objectives. For example, if the typical 
student's previous work has involved related tasks, many elementary 
details may be omitted. This applies to the analysis of the training 
standard and the development of objectives. AFM 50.5, USAF Formal 
Schools Catalog and paragraph 3 of the specialty description in AFM 
36-1 or AFM. 

Reading Ease Score 

no . of words 100 _ 
Average sentence length = ^o. of sentences = "T" " 

Number of syllables =179 

Reading Ease Score = 206.835 - [1.015(20)+ . 846(179)] = 

206.835 - [20.3+ 151.434] 
206. 835 - 171. 734 34. 101 

Human Interest Score 

Personal words = 1 
Personal sentences = 1 

Human Interest Score = 3.635(1) + .314(1) = 3.949 
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APPENDIX D 

How to Perform the FORCAST Analysis 

The FORCAST technique provides a reading grade level based 
on a monosyllabic word count. An example of the technique is demon- 
strated below. The sample reading passage was excerpted from a mili- 
tary guide for instructors. 



An instructor cannot read the mind of a student to 
verify the extent of his understanding. It is only through 
some overt activity on the part of the student that the ex- 
tent of his knowledge or skill can be determined. To say 
that a student will "develop an understanding of Ohm's 
Law" is often used as an example of an obscure objective. 
One person may feel that if the student writes the law it 
indicates understanding. Another may say that the student 
should explain the law. A third may contend that the only 
way the student can satisfy this objective is for him to use 
the formula to solve problems in electrical circuitry. 

Number of one syllable words in a 150 word passage = 76 

Reading Grade Level = 20 - (^o- of one^syllable words^^ 20-7.6 

= 12 
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How to Perform the SMOG Analysis 

The following example demonstrates the procedure for calculat- 
ing a SMOG grade on a passage of 30 sentences. 

Sample Test 
1. GENERAL INFORMATION 

The need for human engineering in the national defense is 
based on the fact that the science of man. and his capacities 
must keep pace with technology if military effectiveness is 
to be achieved. Machines cannot fight alone; they require 
men to operate and maintain them. The task of human en- 
gineering, both generally and as an element of the Person- 
nel Subsystem (PS), is to get the best from man in the oper- 
ation of the highly complex equipment produced by modern 
technology. (Some of the concepts expressed in this and the 
following paragraphs are based on Ref 193.) Human engi- 
neering proceeds on the assumption that the capacities of 
man develop within certain limits and that by adapting the 
design of equipment to these capabilities, a more effective 
system will result. A system that requires extreme, un- 
interrupted concentration over long periods of time would 
not be "human engineered" nor could a system be relied 
upon in combat if it were so complex that it could be main- 
tained only by the engineers and scientists who designed it. 
As used in this handbook, the term "human engineering" is 
not synonymous with human factors. The term "human fac- 
tors" is more comprehensive, covering all biomedical and 
psychosocial considerations applying to man in the system. 
It includes not only human engineering, but also life support, 
personnel selection and training, training equipment, job 
performance aids, and performance measurement and eval- 
uation. Human factors is closely related to and is often 
used as a synonum for PS. Sometimes the total human fac- 
•tors effort (including human engineering and other PS ele- 
ments) is called "bioastronautics" or "life sciences." 
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2. THE PRACTICE OF HUMAN ENGINEERING 



System performance must not be degraded by improper de- 
sign which imposes limitations on the human operator or 
maintenance technican. Consequently, the Air Force Sys- 
tems Command (AFSC) has been assigned responsibility for 
ensuring that new systems are planned and designed so as 
to exploit known human capabilities, but, at the same time, 
not require any performance which exceeds those capabil- 
ities. Carrying out this responsibility constitutes the "prac 
tice" of human engineering in the Air Force context. 



2, 1 Who Does Human Engineering? 

No single professional group is responsible for research in 
human engineering or the application of human engineering 
principles to the design of systems. Engineers, industrial 
engineers, anthropologists, physicians, physiologists, and 
psychologists all participate. In a broad sense, everyone 
concerned with the design or improvement of devices which 
people use or the ways in which they use them is practicing 
human engineering. However, as so often happens in a tech 
nical field, a point is reached where every man can no longe 
be his own specialist. The profession of human engineering 
has emerged to meet this demand. Human engineering draw 
its basic data from several fields of knowledge, among them 
(1) the engineering sciences, (2) experimental psychology, 
(3) physical anthropology, and (4) psychophysiology. The 
profession calls for varied talents and broad training in both 
the physical sciences and the sciences that deal with man. It 
is understandable, therefore, that many human engineering 
specialists are experimental psychologists conversant with 
engineering or engineers with supplementary training in the 
behavioral sciences. 
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2, 2 What Do Human Engineers Do? 

Human engineers conduct research, establish principles, 
and assist in designing and developing system hardware 
that is properly engineered for effective human use. They 
study the human being as a user of equipment and equip- 
ment as a tool for a task. In collaboration with hardware 
engineers and designers, human engineers seek to develop 
new and improved man- equipment interfaces that will sim- 
plify the operator's task and increase the probability of 
mission accomplishment. They seek to achieve displays 
that will most effectively present information to the human 
senses, to obtain the most efficient controls for human 
use, and to provide an optimum work environment. They • 
contribute to the development of such complex man-equip- 
ment systems as combat-information centers, air defense 
systems, space vehicles, and offensive weapon systems. 
In the design of military equipment, human engineers place 
emphasis on efficiency, as measured by the speed and ac- 
curacy of performance, and on the safety and comfort of 
the human operator. ' Because the successful design of 
equipment for human use requires consideration of man's 
basic characteristics, human engineers study man's sen- 
sory capacities, his muscular strength and coordination, 
his body dimensions, his perception and judgment, his 
native skills, his optimum workload, and his requirements 
for comfort, safety, and freedom from environmental stress. 



3. HUMAN ENGINEERING IN THE AIR FORCE 

Within the Air Force, AFSC is responsible for both the de- 
velopment and the application of human engineering know- 
ledge and techniques. 

Sample Calculation 

SMOG Grade = 3 + -s/words of 3 syllables or more in 30 sentences 
= 3 + n/179 = 3 + 13.4 
= 16 
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APPENDIX F 
Glossary of Readability Measures 
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